Depth camera, multi-depth camera system and method of synchronizing the same

ABSTRACT

A depth camera includes a sensor unit receiving a reflected light and in response thereto outputting an electrical sensing signal; and a synchronization information calculation unit calculating a performance index with reference to the sensing signal, and with reference to the performance index, generating synchronization information for synchronizing a demodulation clock for sensing the received reflected light. The sensor unit adjusts the frequency and/or phase of the demodulation clock with reference to the synchronization information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This U.S. non-provisional patent application claims priority under 35U.S.C. §119 of Korean Patent Application No. 10-2012-0027738, filed onMar. 19, 2012, the entire contents of which are hereby incorporated byreference.

BACKGROUND

The present inventive concept herein relates to image sensors, and moreparticularly, to a depth camera for obtaining a three dimensional imageand a method of synchronizing the depth camera.

As image processing technology develops, an interest in the field ofmodeling an object in a three-dimensional image is becoming great.Modeling an object in a three-dimensional image can be applied to avirtual reality movie and to a computer graphic of a video game. Thus,an image modeled in three dimensions is expected to be applied tovarious fields.

Three-dimensional image information includes geometry information andcolor information. The geometry information can be obtained using adepth image. The depth image can be directly obtained using filmingequipment such as a depth camera. The depth image can also be indirectlyobtained using an image process which is a computer vision technology,without using a depth camera.

In the method of obtaining the depth image using the depth camera, amethod of measuring a time taken until an emitted light returns after itis reflected by an object is widely used. The time is referred to as“time of flight (ToF)”. To realize a three dimensional image in a morerealistic angle, a multi-ToF method using a plurality of depth camerasis needed. To realize a three dimensional image using a plurality ofdepth cameras, synchronization of the plurality of depth cameras becomesan important issue. A trigger for initialization is periodically appliedto a plurality of depth cameras to perform synchronization. However, inthis method, a frame rate of the three-dimensional image is reduced anda deviation may occur in synchronization because of a distancedifference between a host providing a trigger and the cameras. If usingemitting lights having different frequencies or pulse widths, a deptherror may increase due to interference between the depth cameras.

Thus, to obtain a three-dimensional image having high resolution andhigh performance in a multi-ToF method, a technology that can performefficient synchronization of the depth cameras is needed.

SUMMARY

Embodiments of the inventive concept provide a depth camera. The depthcamera may include a sensor unit configured to receive a reflected lightand in response thereto to output an electrical sensing signal; and asynchronization information calculation unit configured to calculate aperformance index with reference to the electrical sensing signal, andwith reference to the performance index to generate synchronizationinformation for synchronization of a demodulation clock for sensing thereceived reflected light, wherein the sensor unit is configured toadjust at least one of a frequency and a phase of the demodulation clockwith reference to the synchronization information.

Embodiments of the inventive concept also provide a method ofsynchronizing a depth camera receiving a light reflected from a targetobject to generate depth information. The method may include sensing areflected light being received, and in response thereto outputting anelectrical sensing signal; calculating a performance index of the depthcamera with reference to the electrical sensing signal; and withreference to the performance index, adjusting at least one of afrequency and a phase of a demodulation clock for sensing the receivedreflected light.

Embodiments of the inventive concept also provide a multi depth camerasystem. The multi depth camera system may include a plurality of depthcameras including at least one reference camera; and a host configuredto receive depth information for one or more areas of a target objectfrom each of the plurality of depth cameras and in response thereto togenerate a three-dimensional image of the one or more areas of thetarget object, wherein at least one of the depth cameras includes asensor unit that senses reflected light received from the object inresponse to a demodulation clock of the at least one depth camera, andwherein the at least one depth camera synchronizes the demodulationclock to the reference camera in response to the received reflectedlight.

BRIEF DESCRIPTION OF THE FIGURES

Preferred embodiments of the inventive concept will be described belowin more detail with reference to the accompanying drawings. Theembodiments of the inventive concept may, however, be embodied indifferent forms and should not be constructed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the inventive concept to those skilled in the art.Like numbers refer to like elements throughout.

FIG. 1 is a drawing illustrating a multi-depth camera.

FIG. 2 is a block diagram illustrating one embodiment of a depth camerain accordance with the inventive concept.

FIG. 3 is a block diagram illustrating in more detail embodiments of asensor unit and a synchronization information calculation unit inaccordance with the inventive concept.

FIG. 4 is a graph illustrating a relationship between an emitting lightand a reflected light in accordance with some embodiments of theinventive concept.

FIG. 5 is a graph illustrating a waveform of demodulation contrast (DC)as an example of a performance index of a depth camera.

FIG. 6 is a flow chart illustrating an embodiment of a phase and/orfrequency synchronization method performed in a depth camera inaccordance with the inventive concept.

FIG. 7 is a flow chart illustrating some other embodiments ofsynchronization methods of the depth camera of FIG. 2.

FIG. 8 is a drawing illustrating a three-dimensional camera system usinga plurality of the depth cameras of FIG. 2.

FIG. 9 is a block diagram illustrating some other embodiments of thedepth camera in accordance with the inventive concept.

FIG. 10 is a block diagram illustrating in more detail other embodimentsof a sensor unit and a synchronization information calculation unit inaccordance with the inventive concept.

FIG. 11 is a drawing illustrating a three-dimensional camera systemusing a plurality of the depth cameras of FIG. 9.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of inventive concepts will be described more fullyhereinafter with reference to the accompanying drawings, in whichembodiments of the invention are shown. This inventive concept may,however, be embodied in many different forms and should not be construedas limited to the embodiments set forth herein. Rather, theseembodiments are provided so that this disclosure will be thorough andcomplete, and will fully convey the scope of the inventive concept tothose skilled in the art. In the drawings, the size and relative sizesof layers and regions may be exaggerated for clarity. Like numbers referto like elements throughout.

As used herein, the term “and/or” includes any and all combinations ofone or more of the associated listed items and may be abbreviated as“/”. It will be understood that when an element is referred to as being“connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. It will be further understood that the terms “comprises” and/or“comprising,” or “includes” and/or “including” when used in thisspecification, specify the presence of stated features, regions,integers, steps, operations, elements, and/or components, but do notpreclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components, and/orgroups thereof.

FIG. 1 is a drawing illustrating a multi-depth camera for providing athree-dimensional image as an illustration. Referring to FIG. 1, themulti-depth camera includes a plurality of depth cameras 10 and 20measuring depth information of an object 30. Although not illustrated inthe drawing, a host may be included which composes geometry informationprovided from each of depth cameras 10 and 20 to form athree-dimensional image.

First depth camera 10 can sense depth information on areas 31 and 32 ofobject 30. However, first depth camera 10 cannot sense depth informationon an area 33. Area 33 is called an occlusion of the first depth camera10. Second depth camera 20 can sense depth information on areas 31 and33 of object 30. However, second depth camera 20 cannot sense depthinformation on area 32. Area 32 is called an occlusion of second depthcamera 20.

The depth information that can be obtained at the same time bymulti-depth cameras 10 and 20 is geometry information regarding area 31.Three-dimensional image information regarding area 31 can be created bycombining depth information obtained from multi-depth cameras 10 and 20.However, three-dimensional image information regarding areas 32 and 33cannot be generated. Thus, if more depth cameras are employed, depthinformation on a wider area can be obtained. The depth informationobtained from a plurality of depth cameras should be informationmeasured at the same time. If synchronization of depth cameras isincomplete, geometry information cannot be created for an accuratethree-dimensional image.

According to some embodiments of the inventive concept, each of thedepth cameras can be initialized or synchronized to a frequency and/orphase of the reflected light. Thus, as compared with the case that eachof the depth cameras is synchronized by an external single trigger, theeffect of a distance deviation between the depth cameras and the host,and interference of the reflected light, can be reduced.

FIG. 2 is a block diagram illustrating one embodiment of a depth camerain accordance with the inventive concept. Referring to FIG. 2, the depthcamera 100 includes a lens 105, a sensor unit 110, a synchronizationinformation calculation unit 120 and an illuminator 130. A frequencyand/or a phase of a clock can be synchronized with the receivedreflected light by a feedback structure of sensor unit 110 andsynchronization information calculation unit 120.

Lens 105 focuses reflected light being received by depth camera 100 totransmit the focused reflected light to sensor unit 110. Lens 105 isillustrated to be one convex lens but the inventive concept is notlimited thereto. Lens 105 may be a convex lens, a concave lens orcombination thereof. Lens 105 can be replaced with various opticalstructures.

Sensor unit 110 receives a reflected light (RL) from the target object(TO). Sensor unit 110 senses the reflected light (RL) focused throughlens 105, beneficially using a plurality of sensor arrays. Sensor unit110 accumulates electrical charges corresponding to the strength of thereflected light (RL). Sensor unit 110 outputs an electrical sensingsignal Sout corresponding to the accumulated electrical charges. Sensorunit 110 transmits the sensing signal Sout to synchronizationinformation calculation unit 120.

Sensor unit 110 may include a demodulation clock Demod_CLK (see, e.g.,FIG. 3) for gating a cell of a sensor array of sensor unit 110, and aclock generator for generating a modulation clock Mod_CLK to controllight emitted by illuminator 130.

Synchronization information calculation unit 120 measures performance ofdepth camera 100 with reference to the sensing signal Sout provided fromsensor unit 110. For example, synchronization information calculationunit 120 can calculate a demodulation contrast or a depth error as aperformance index with reference to the sensing signal Sout providedfrom sensor unit 110. Synchronization information calculation unit 120can determine a control direction of the frequency and/or phase of thedemodulation clock with reference to the calculated performance index.Synchronization information calculation unit 120 provides the calculatedcontrol information of frequency and/or phase to sensor unit 110 assynchronization information (SI).

Sensor unit 110 controls a phase of the modulation and/or demodulationclock being generated with reference to the synchronization information(SI) which feeds back from synchronization information calculation unit120. If it needs to delay a phase of the modulation and/or demodulationclock for optimum performance, sensor unit 110 delays a phase of a clockgenerating unit. A phase of the modulation clock being provided toilluminator 130 and a phase of the demodulation clock being provided asa gating signal of the sensor array included in the sensor unit 110 arealso delayed.

Illuminator 130 emits an emitted light (EL) into the target object (TO)according to the modulation clock Mod_CLK being provided from sensorunit 110. In particular, the emitted light (EL) from illuminator 130 maybe modulated in accordance with the modulation clock MOD_CLK. Forexample, illuminator 130 can emit a series of light pulses correspondingto the frequency of the modulation clock Mod_CLK. Illuminator 130 can berealized by a light emitting diode (LED) array or a laser device. Apulse of light emitted from illuminator 130 may be variously realized tobe infrared light, ultraviolet light, visible light, an ultrasonic wave,etc.

Depth camera 100 in accordance with some embodiments of the inventiveconcept can determine a phase and/or frequency of amodulation/demodulation clock that can provide maximum performance withreference to the reflected light (RL). Thus, depth camera 100 candetermine a gating clock of the light sensor and a frequency and/orphase of the modulation/demodulation clock for generating the emittedlight.

FIG. 3 is a block diagram illustrating in more detail embodiments of asensor unit 110 and a synchronization information calculation unit 120according to the inventive concept. Referring to FIG. 3, sensor unit 110includes a clock generator 112, a sensor array 114 and a clockcontroller 116. Synchronization information calculation unit 120includes a measuring unit 122 and a decision unit 124.

Clock generator 112 generates the modulation clock Mod_CLK beingprovided to illuminator 130 and the demodulation clock Demod_CLK beingprovided to sensor array 114 as a gating signal. Clock generator 112 cancontrol a phase and/or frequency of the modulation/demodulation clockgenerated under control of clock controller 116.

Sensor array 114 senses the reflected light (RL) in synchronization withthe demodulation clock Demod_CLK being provided from clock generator112. A plurality of sensor pixels included in sensor array 114 sense thereflected light (RL) in synchronization with the demodulation clockDemod_CLK. For example, when a waveform of demodulation clock Demod_CLKis high, the plurality of sensor pixels may receive the reflected light(RL) to accumulate charges. Sensor array 114 can be embodied in aphotodiode array or a photo gate array wherein the plurality of sensorpixels is two-dimensionally arranged.

Clock controller 116 can control a phase and/or frequency of clockgenerator 112 with reference to synchronization information (SI)provided from synchronization information calculation unit 120. Clockcontroller 116 can control the phase of the modulation clock Mod_CLKand/or demodulation clock Demod_CLK with reference to thesynchronization information (SI). Clock controller 116 can also controlthe frequency of the modulation clock Mod_CLK and/or demodulation clockDemod_CLK with reference to the synchronization information (SI).

The synchronization information (SI) can indicate that a phase of themodulation clock Mod_CLK and/or demodulation clock Demod_CLK beinggenerated is delayed as compared with a reference phase. Clockcontroller 116 can control clock generator 112 so that the phase of themodulation clock Mod_CLK and/or demodulation clock Demod_CLK beinggenerated shifts in an opposite direction to the delay. In the oppositecase, clock controller 116 can delay a phase of clock generator 112.Clock controller 116 can control the frequency of modulation clockMod_CLK and/or demodulation clock Demod_CLK with reference to thesynchronization information (SI).

Sensor unit 110 provides a signal Sout sensed in sensor array 114 tosynchronization information calculation unit 120 and receives thesynchronization information (SI) which is fed back from synchronizationinformation calculation unit 120. Sensor unit 110 can adjust a phaseand/or frequency of the modulation/demodulation clock with reference tothe synchronization information (SI). Clock generator 112 is illustratedto be a constituent element of sensor unit 110 but the inventive conceptis not limited thereto. That is, in some embodiments clock generator 112may be located outside the sensor unit 110 to provide the modulationclock and the demodulation clock to sensor unit 110 and illuminator 130.

Synchronization information calculation unit 120 includes a measuringunit 122 and a decision unit 124. Measuring unit 122 measures aperformance index of depth camera 100 with reference to a sensing signalSout being provided from sense array 114. Measuring unit 122 cancalculate a demodulation contrast (DC) from the sensing signal Soutbeing provided by sensing the reflected light (RL). The demodulationcontrast (DC) represents a degree of precision of phase shift in eachsensor pixel. If a phase shifts in a direction such that thedemodulation contrast increases, the degree of precision of sensing thereceived reflected light increases and thereby performance is improved.Measuring unit 122 can calculate a depth error from the sensing signalSout by sensing the reflected light (RL).

Decision unit 124 determines a compensation direction and a magnitude ofa frequency and/or phase of clock generator 112 with reference to aperformance index being provided from measuring unit 122. For example,decision unit 124 can determine a direction for controlling the phasesuch that the magnitude of the demodulation contrast DC increases.Decision unit 124 determines a direction of phase control, and providesthis to clock controller 116 as synchronization information (SI). Thedecision operation of decision unit 124 can be applied to a frequency ofthe modulation clock Mod_CLK and/or the demodulation clock Demod_CLK.

A synchronization process of depth camera 100 of the inventive conceptwas described through a feedback structure of sensor unit 110 andsynchronization information calculation unit 120. The depth camera canbe synchronized with a reference emitted light by sensing the reflectedlight (RL). The reference emitting light may be provided from a separatedepth camera located outside depth camera 100, or may be provided from aseparate light emitting means.

FIG. 4 is a graph illustrating a relationship between an emitting light(EL) and a reflected light (RL) in accordance with some embodiments ofthe inventive concept. Referring to FIG. 4, when the emitting light (EL)having a sinusoidal wave shape is emitted to an object, a waveform ofthe reflected light (RL) received by sensor unit 114 is as illustrated.In FIG. 4, the emitting light (EL) is represented by a curve C1 and thereflected light (RL) is represented by a curve C2.

The emitting light C1 and the reflected light C2 have a phase differenceΦ between them. In particular, the reflected light C2 is received to bedelayed by a phase difference Φ as compared with the emitting light C1.When the emitting light C1 and the reflected light C2 have the samefrequency as each other, then the reflected light C2 represents atime-of-flight (ToF) corresponding to the phase difference Φ. A sensorpixel of sensor unit 110 may output depth information corresponding tothe phase difference Φ.

The reflected light C2 is defined by an offset (B) representing a directcurrent value, and an amplitude (A) of the waveform. Amplitude (A) ofthe reflected light C2 can be obtained through amplitudes A0, A1, A2 andA3 corresponding to a plurality of sampling times T0, T1, T2 and T3during one period. Amplitude (A) can be obtained by equation 1.

$\begin{matrix}{A = \frac{\sqrt{\left( {A_{3} - A_{1}} \right)^{2} + \left( {A_{2} - A_{0}} \right)^{2}}}{2}} & \left\lbrack {{EQN}\mspace{14mu} 1} \right\rbrack\end{matrix}$

Offset (B) can be calculated by equation 2.

$\begin{matrix}{B = \frac{A_{0} + A_{1} + A_{2} + A_{3}}{4}} & \left\lbrack {{EQN}\mspace{14mu} 2} \right\rbrack\end{matrix}$

The phase difference Φ and the measured depth are represented byequations 3 and 4 respectively.

$\begin{matrix}{\Phi = {{\tan^{- 1}\left( \frac{A_{3} - A_{1}}{A_{0} - A_{2}} \right)} + \pi}} & \left\lbrack {{EQN}\mspace{14mu} 3} \right\rbrack \\{{Depth} = {\frac{c}{4\pi \; f}\Phi}} & \left\lbrack {{EQN}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Here, c is the speed of light, f is the frequency of the reflectedlight, and (I) is the phase difference between the emitting light andthe reflected light.

Various performance indexes of the depth camera can be obtained withrespect to the reflected light C2 defined by the above-mentionedparameters. Examples of the performance index are a demodulationcontrast (DC), a depth error, etc.

The demodulation contrast (DC), which is an example of a performanceindex, can be calculated by equation 5.

$\begin{matrix}{{DC} = \frac{A}{B}} & \left\lbrack {{EQN}\mspace{14mu} 5} \right\rbrack\end{matrix}$

Here, A is the measured amplitude and B is the measured offset.

The depth error (DE), which is another example of a performance index,can be calculated by equation 6.

$\begin{matrix}{{DE} = \frac{{VAR}({Depth})}{Distance}} & \left\lbrack {{EQN}\mspace{14mu} 6} \right\rbrack\end{matrix}$

Here, “Distance” is a distance between a target object and the depthcamera.

Measuring unit 122 can obtain those indices using the parameters of thereflected light C2. Decision unit 124 can determine a shift direction ofphase, or an addition and subtraction of frequency, for the reflectedlight to be synchronized with the reference emitting light, withreference to the performance index.

FIG. 5 is a graph illustrating a waveform of the demodulation contrast(DC) as an example of a performance index of a depth camera. Referringto FIG. 5, a value of the demodulation contrast (DC) varies depending ona phase difference between the emitting light and the reflected light.Decision unit 124 can determine a control direction of the phase and/orfrequency with reference to the demodulation contrast (DC) calculatedthrough the reflected light that is currently being sensed or detected.

The demodulation contrast (DC) is a parameter which can represent themeasurement degree of precision of the phase shift. The greater thedemodulation contrast (DC) is, the higher the degree of precision of themeasurement by the sensor pixel is. In the case of using multi cameras,as the synchronization of the depth cameras with respect to a lightsource becomes more accurate, the demodulation contrast (DC) becomesgreater. When using two depth cameras, the demodulation contrast (DC) ismaximized when a phase difference between the emitting lights of thedepth cameras is zero. The demodulation contrast (DC) is minimized whena phase difference between the emitting lights of the depth cameras is180° (or π).

Thus, decision unit 124 can provide the synchronization information (SI)indicating the direction that a phase difference is reduced consideringthe transition of the change. For example, with reference to FIG. 5, ifthe level of the demodulation contrast (DC) currently measured is DC1,decision unit 124 can generate the synchronization information (SI) soas to increase the phase of the emitting light of the depth camera. Onthe other hand, if the level of the demodulation contrast (DC) currentlymeasured is DC2, decision unit 124 can generate the synchronizationinformation (SI) so as to decrease the phase of the emitting light ofthe depth camera.

FIG. 6 is a flow chart illustrating a phase and/or frequencysynchronization method performed in a depth camera of the inventiveconcept. Referring to FIGS. 3 and 6, the depth camera in accordance withsome embodiments of the inventive concept can control the phase and/orfrequency of a demodulation clock being provided to sensor unit 110and/or a modulation clock being provided to the illuminator 130 withreference to a performance index calculated through a measurement of thereflected light (RL).

In a step S110, sensor unit 110 of depth camera 100 senses the reflectedlight (RL) which is currently being received. By the reflected light(RL) being currently received, each sensor pixel of sensor array 114 issynchronized with the demodulation clock Demod_CLK to accumulatecharges. A pulse light signal of the reflected light (RL) of the levelcorresponding to the accumulated charges is provided to measuring unit122 as an electrical sensing signal Sout.

In a step S120, measuring unit 122 calculates a performance index fromthe sensing signal Sout being output from sensor unit 110. For example,measuring unit 122 may calculate a demodulation contrast (DC) or a deptherror (DE) from the sensing signal Sout. Measuring unit 122 transmitsthe calculated performance index to decision unit 124.

In a step S130, decision unit 124 determines whether adding orsubtracting phase and/or frequency with respect to the current phaseand/or frequency of the modulation clock and/or demodulation clock canincrease performance. For example, decision unit 124 may generatesynchronization information (SI) so that the modulation clock Mod_CLK ofthe emitting light (EL) has a phase which maximizes the demodulationcontrast (DC).

In a step S140, clock controller 116 controls a phase and/or frequencyof clock generator 112 with reference to synchronization information(SI). In particular, the frequency and/or phase of the modulation and/ordemodulation clocks of the depth camera are controlled to besynchronized with a reference frequency and/or phase of the depthcamera.

According to the processes described above, depth camera 100 cansynchronize the frequency and/or phase of its modulation and/ordemodulation clocks for sensing or emitting light, with clocks of otherdepth cameras of a multi depth camera.

FIG. 7 is a flow chart illustrating some other embodiments ofsynchronization method of depth camera of FIG. 2. Referring to FIGS. 3and 7, depth camera 100 performs a plurality of synchronization loops atan initializing operation or reset operation to be fixed to the optimumfrequency or phase.

In a step S210, sensor unit 110 of depth camera 100 senses the reflectedlight (RL) currently being received. Each sensor pixel of sensor array114 which senses or detects the reflected light (RL) currently beingreceived is synchronized with the demodulation clock Demod_CLK toaccumulate charges. A light pulse signal of the reflected light (RL)having a level corresponding to the accumulated charges is provided tomeasuring unit 122 as the electrical sensing signal Sout.

In a step S220, measuring unit 122 calculates a performance index fromthe sensing signal Sout being output from sensor unit 110. Measuringunit 122 calculates a demodulation contrast (DC) or a depth error from asensing signal Sout. The measuring unit 122 transmits the calculatedperformance index to decision unit 124.

In a step S230, decision unit 124 determines a direction of adding orsubtracting to a current frequency and/or phase which may increase theperformance of sensor unit 110. For example, decision unit 124 cangenerate synchronization information (SI) so as to generate a modulationclock Mod_CLK and/or demodulation clock Demod_CLK so that a value ofdemodulation contrast (DC) is maximized.

In a step S240, with reference to the determined performance index, itis determined whether the modulation clock Mod_CLK and/or demodulationclock Demod_CLK of depth camera 100 is synchronized with an optimumfrequency or phase. If it is determined that the modulation clockMod_CLK and/or demodulation clock Demod_CLK of depth camera 100 issynchronized with the optimum frequency or phase, a process goes to astep S260 for locking the modulation clock Mod_CLK and/or demodulationclock Demod_CLK at the synchronized frequency or phase. If it isdetermined that the modulation clock Mod_CLK and/or demodulation clockDemod_CLK of depth camera 100 is not synchronized with an optimumfrequency or phase, then the process goes to a step S250.

In step S250, clock controller 116 adjusts the phase and/or frequency ofclock generator 112 with reference to synchronization information (SI)being provided from decision unit 124. Then the process returns to stepS210 for sensing the reflected light (RL) under conditions of anadjusted modulation clock Mod_CLK and/or demodulation clock Demod_CLK.

In step S260, clock controller 116 sets up clock generator 112 so as tooutput a modulation clock Mod_CLK and/or a demodulation clock Demod_CLKof the synchronized frequency or phase.

The steps S210, S220, S230, S240 and S250 constitute an operating loopfor adjusting a frequency and/or phase. That operating loop repeatsuntil a modulation and/or demodulation clock of depth camera 100 issynchronized with an emitting light (EL) which becomes a reference of amulti camera system.

FIG. 8 is a drawing illustrating a three-dimensional camera system u200sing the plurality of the depth cameras which are illustrated in FIG. 2.Multi camera system 200 includes a multi camera 210 and a host 220.

Multi camera 210 includes a plurality of depth cameras 212, 214 and 216.Each depth camera includes an illuminator and a sensor unit. Thus, eachdepth camera can perform a frequency and/or phase synchronization withreference to an emitting light (EL) being independently received. Theplurality of depth cameras 212, 214 and 216 includes a reference camera216.

Reference camera 216 emits an emitting light (EL) generated by amodulation clock of a locked frequency or phase into a target object(TO). Reference camera 216 receives a reflected light (RL) at aspecified location to generate depth information of the target object(TO). The depth information generated by reference camera 216 isprovided to host 220. That is, reference camera 216 operates with afixed value without adjusting a modulation or demodulation clock.

Depth cameras 212 and 214 other than reference camera 216 perform afrequency and/or phase synchronization of a modulation clock and/or ademodulation clock in response to the reflected light being received.Thus, depth cameras 212 and 214 each perform a frequency and/or phasesynchronization so that the frequency and/or phase of their modulationand/or demodulation clocks are synchronized with reference camera 216.That is, the modulation clock and/or demodulation clock of depth cameras212 and 214 can be synchronized with the frequency and/or phase ofmodulation of the emitting light (EL) emitted from reference camera 216.

According to the synchronization process, an electrical delay does notoccur as compared with a case of being synchronized through a trigger byhost 220. Even though a slight frequency or phase shift of referencecamera 216 occurs, all the depth cameras included in multi camera system200 can be organically synchronized by an adaptive synchronizationoperation of depth cameras 212 and 214. Thus, it is possible to obtainaccurate configuration information on the target object.

Host 220 processes depth information provided from depth cameras 212,214 and 216. Host 220 can recreate a three-dimensional image of a targetobject at multiple positions. Host 220 can select reference camera 216defining a frequency reference and a phase reference among plurality ofdepth cameras 212, 214 and 216. A depth camera selected as the referencecamera calculates a performance index according to selection informationbut does not perform an operation adjusting the phase and/or frequencyof its clock(s).

FIG. 9 is a block diagram illustrating some other embodiments of thedepth camera of the inventive concept. Referring to FIG. 9, depth camera300 includes a lens 305, a sensor unit 310 and a synchronizationinformation calculation unit 320. Depth camera 300 does not include anilluminator. Thus, only a demodulation clock Demod_CLK being provided tosensor unit 310 by synchronization is adjusted with reference toreflected light which is sensed or detected by sensor unit 310.

Lens 305 focuses the reflected light being received by depth camera 300to transmit the focused reflected light to sensor unit 310. Lens 305 isillustrated to be one convex lens but the inventive concept is notlimited thereto. Lens 305 can be replaced with various light-receivingmeans or light focusing means having transparent quality of thematerial.

Sensor unit 310 receives the reflected light (RL) from a target object(TO). Sensor unit 310 senses the reflected light (RL) focused throughlens 305 using a plurality of sensor array. Sensor unit 310 accumulateselectrical charges corresponding to a strength of the reflected light(RL) which is being received. Sensor unit 310 can obtain a waveform,frequency and/or phase information of the reflected light (RL) accordingto the accumulated electrical charges. Sensor unit 310 transmits theobtained information to synchronization information calculation unit 320as a sensing signal Sout.

A clock generator for creating a demodulation clock Demod_CLK to gate apixel of sensor array is included in sensor unit 310.

Synchronization information calculation unit 320 measures performance ofdepth camera 300 with reference to the sensing signal Sout beingprovided from sensor unit 310. Synchronization information calculationunit 320 can calculate a demodulation contrast or a depth error as aperformance index with reference to the sensing signal Sout.Synchronization information calculation unit 320 can determine adirection of a phase shift of the demodulation clock Demod_CLK forimproving performance with reference to the calculated performanceindex. Synchronization information calculation unit 320 provides thedetermined synchronization information to sensor unit 310.

Sensor unit 310 adjusts the phase of the demodulation clock Demod_CLKbeing generated with reference to the synchronization information (SI)which feeds back from synchronization information calculation unit 320.In the case that a phase of the clock signal needs to be delayed foroptimum performance, then sensor unit 310 delays the phase of the clockgenerator. When the phase of the clock generator is delayed, then thephase of the demodulation clock Demod_CLK provided as a gating signal ofsensor array included in sensor unit 310 is also delayed.

Depth camera 300 can determine a phase of frequency of clock that canprovide the maximum performance with reference to a reflected light(RL). Thus, a frequency and/or phase of demodulation clock Demod_CLK forgenerating a gating clock of optical sensor can be determined withreference to an optical sensor.

FIG. 10 is a block diagram illustrating a sensor unit and asynchronization information calculation unit of the inventive concept inmore detail. Referring to FIG. 10, sensor unit 310 includes a clockgenerator 312, a sensor array 314 and a clock controller 316.Synchronization information calculation unit 320 includes a measuringunit 322 and a decision unit 324.

Sensor unit 310 includes a clock generator 312, a sensor array 314 and aclock controller 316. Clock generator 312 generates a demodulation clockDemod_CLK being provided as a sensing clock of sensor array 314. Undercontrol of clock controller 316, clock generator 312 can control thephase and/or frequency of the generated clock signal Demod_CLK.

Sensor array 314 senses a reflected light (RL) in synchronization withthe demodulation clock Demod_CLK provided from clock generator 312. Aplurality of sensor pixels included in sensor array 314 sense thereflected light (RL) in synchronization with the demodulation clockDemod_CLK. For example, when a waveform of the demodulation clockDemod_CLK is high, the plurality of sensor pixels may receive thereflected light (RL) to accumulate charges. Sensor array 314 can beembodied in a photodiode array or a photo gate array wherein theplurality of sensor pixels is two-dimensionally arranged.

Clock controller 316 can control the phase and/or frequency of clockgenerator 312 with reference to synchronization information (SI)provided from synchronization information calculation unit 320. Clockcontroller 316 can control the phase of the demodulation clock Demod_CLKwith reference to the synchronization information (SI). Clock controller316 can also adjust the frequency of demodulation clock Demod_CLK withreference to the synchronization information (SI).

Sensor unit 310 provides a signal sensed in sensor array 314 tosynchronization information calculation unit 320 and can adjust thephase and/or frequency of the clock with reference to thesynchronization information (SI) which feeds back from synchronizationinformation calculation unit 320. Clock generator 312 is illustrated tobe a constituent element of sensor unit 310 but the inventive concept isnot limited thereto. That is, in some embodiments, clock generator 312may be located outside sensor unit 310 to provide the demodulation clockto sensor unit 310.

The synchronization information calculation unit 320 includes ameasuring unit 322 and a decision unit 324. Measuring unit 322 measuresa performance index of depth camera 300 with reference to the sensingsingle Sout provided from sensor array 314. For example, measuring unit322 may calculate a demodulation contrast (DC) or depth error (DE) froma sensing signal Sout provided by sensing the reflected light (RL).

For example, with reference to a value of the demodulation contrast(DC), decision unit 324 determines a phase shift in a direction suchthat an amplitude of the demodulation contrast (DC) increases. Decisionunit 324 determines a direction of the phase shift to provide it toclock controller 316 as synchronization information (SI).

A method of controlling a clock that can improve performance through afeedback structure of synchronization information calculation unit 320has been described above.

FIG. 11 is a drawing illustrating a three-dimensional camera systemusing the plurality of depth cameras which are illustrated in FIG. 9.Referring to FIG. 11, multi camera system 400 of the inventive conceptincludes a multi camera 410 and a host 420.

Multi camera 410 includes a plurality of depth cameras 412, 414 and 416.Each depth camera includes a sensor unit. Beneficially, none of thedepth cameras includes an illuminator. Only one illuminator 411 isincluded in multi camera 410.

In the above-mentioned structure, reference camera 416 is synchronizedwith illuminator 411. In that case, a gating of illuminator 411 and thepixel array of reference camera 416 is synchronized. And then, depthcameras 412 and 414 are synchronized by a light emitted from illuminator411 of which the phase and the frequency are synchronized by referencecamera 416. Thus, frequencies and phases of depth cameras 412 and 414are synchronized with reference camera 416.

According to the synchronization process, an electrical delay does notoccur as compared with a case of being synchronized through a trigger byhost 420. Even though a slight frequency or phase shift of referencecamera 416 occurs, all the depth cameras included in multi camera system400 can be organically synchronized by an adaptive synchronizationoperation of depth cameras 412 and 414. Thus, it is possible to obtainaccurate configuration information on the target object.

Host 420 processes depth information provided from depth cameras 412,414 and 416. Host 420 can recreate a three-dimensional image of a targetobject at multiple positions. Host 420 can select reference camera 416defining the frequency reference and the phase reference among theplurality of depth cameras 412, 414 and 416. A depth camera selected asthe reference camera calculates a performance index according toselection information, but does not perform an operation adjusting thephase and/or frequency of its clock(s).

The camera device or the camera system according to the inventiveconcept can be mounted using various types of packages. The flash memorydevice and/or the memory controller in accordance with the inventiveconcept can be mounted by various types of packages such as PoP (packageon package), ball grid array (BGA), chip scale package (CSP), plasticleaded chip carrier (PLCC), plastic dual in-line package (PDIP), die inwaffle pack, die in wafer form, chip on board (COB), ceramic dualin-line package (CERDIP), plastic metric quad flat pack (MQFP), thinquad flat pack (TQFP), small outline (SOIC), shrink small outlinepackage (SSOP), thin small outline (TSOP), thin quad flatpack (TQFP),system in package (SIP), multi chip package (MCP), wafer-levelfabricated package (WFP) and wafer-level processed stack package (WSP).

According to the inventive concept, through an efficient synchronizationof multi depth camera, an accuracy of depth image can be improved, anocclusion of image can be reduced and a power of light source can bereduced.

The foregoing is illustrative of the inventive concept and is not to beconstrued as limiting thereof. Although a few embodiments of theinventive concept have been described, those skilled in the art willreadily appreciate that many modifications are possible in theembodiments without materially departing from the novel teachings andadvantages of the present invention. Accordingly, all such modificationsare intended to be included within the scope of the present invention asdefined in the claims. The present invention is defined by the followingclaims, with equivalents of the claims to be included therein.

What is claimed is:
 1. A depth camera comprising: a sensor unitconfigured to receive a reflected light and in response thereto tooutput an electrical sensing signal; and a synchronization informationcalculation unit configured to calculate a performance index withreference to the electrical sensing signal, and with reference to theperformance index to generate synchronization information forsynchronization of a demodulation clock for sensing the receivedreflected light, wherein the sensor unit is configured to adjust atleast one of a frequency and a phase of the demodulation clock withreference to the synchronization information.
 2. The depth camera ofclaim 1, wherein the sensor unit adjusts a phase of the demodulationclock with reference to the synchronization information so that asensing degree of precision with respect to the received reflected lightis maximized.
 3. The depth camera of claim 2, wherein the performanceindex comprises a demodulation contrast representing a ratio of ameasured amplitude of the received reflected light to a measured offsetof the received reflected light.
 4. The depth camera of claim 2, whereinthe performance index comprises a depth error of the received reflectedlight.
 5. The depth camera of claim 1, wherein the sensor unitcomprises: a sensor array configured to sense the received reflectedlight in synchronization with the demodulation clock and to provide theelectrical sensing signal to the synchronization information calculationunit; a clock generator configured to provide the demodulation clock tothe sensor array; and a clock controller configured to control the clockgenerator with reference to the synchronization information.
 6. Thedepth camera of claim 5, further comprising a light emitting unitconfigured to emit an emitting light in synchronization with amodulation clock provided from the clock generator.
 7. The depth cameraof claim 1, wherein the synchronization information calculation unitcomprises: a measuring unit configured to calculate the performanceindex with reference to the electrical sensing signal; and a decisionunit configured to determine a compensation direction of at least one ofthe frequency and the phase of the demodulation clock with reference tothe performance index.
 8. A method of synchronizing a depth camerareceiving a light reflected from a target object to generate depthinformation, the method comprising: sensing a reflected light beingreceived, and in response thereto outputting an electrical sensingsignal; calculating a performance index of the depth camera withreference to the electrical sensing signal; and with reference to theperformance index, adjusting at least one of a frequency and a phase ofa demodulation clock for sensing the received reflected light.
 9. Themethod of claim 8, wherein the performance index comprises one of: (1) ademodulation contrast representing a degree of precision of sensing thereceived reflected light, and (2) depth error information representingan accuracy of the electrical sensing signal.
 10. The method of claim 9,wherein in the step of adjusting the at least one of the frequency andthe phase of the demodulation clock, the demodulation clock is adjustedso as to maximize the demodulation contrast.
 11. The method of claim 10,wherein in the step of adjusting the at least one of the frequency andthe phase of the demodulation clock, the demodulation clock is adjustedso as to minimize the depth error.
 12. The method of claim 8, furthercomprising emitting an emitting light onto the target object, theemitting light being optically modulated by a modulation clockcorresponding to the demodulation clock.
 13. The method of claim 8,further comprising determining whether the demodulation clock issynchronized for optimum sensing of the reflected light.
 14. The methodof claim 13, wherein when it is determined that demodulation clock issynchronized for optimum sensing, then at least one of the frequency andphase of the demodulation clock is locked.
 15. A multi depth camerasystem, comprising: a plurality of depth cameras including at least onereference camera; and a host configured to receive depth information forone or more areas of a target object from each of the plurality of depthcameras and in response thereto to generate a three-dimensional image ofthe one or more areas of the target object, wherein at least one of thedepth cameras includes a sensor unit that senses reflected lightreceived from the object in response to a demodulation clock of the atleast one depth camera, and wherein the at least one depth camerasynchronizes the demodulation clock to the reference camera in responseto the received reflected light.
 16. The multi depth camera system ofclaim 15, further comprising an illuminator external to the referencecamera, wherein the illuminator is configured to emit light onto theobject, and wherein the illuminator is synchronized with the referencecamera.
 17. The multi depth camera system of claim 15, wherein the hostis configured to select which of the plurality of cameras is thereference camera.
 18. The multi depth camera system of claim 15, whereinthe sensor unit is configured to output an electrical sensing signal inresponse to the received reflected light, and wherein the at least onedepth camera further comprises: a synchronization informationcalculation unit configured to calculate a performance index withreference to the electrical sensing signal, and with reference to theperformance index to generate synchronization information forsynchronization of the demodulation clock; a clock generator configuredto provide the demodulation clock to the sensor array; and a clockcontroller configured to control the clock generator with reference tothe synchronization information.
 19. The multi depth camera system ofclaim 18, wherein the performance index comprises a demodulationcontrast representing a ratio of a measured amplitude of the reflectedlight to a measured offset of the reflected light.
 20. The multi depthcamera system of claim 18, wherein the performance index comprises adepth error of the received reflected light.